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5 METHOD FOR OBTAINING HIGH-RESOLUTION PERFORMANCE 

6 FROM A SINGLE-CHIP COLOR IMAGE SENSOR 

7 
8 

9 Background of the Invention 

10 This invention relates to machine vision, and particularly to applications of 
n machine vision that exploit color imaging. 

n Color imaging, and other forms of multi-spectral imaging (e.g., imaging that 

u 

K includes near infra-red electromagnetic energy) is useful for facilitating various 

p machine vision tasks, such as alignment and inspection. For example, see US Pat. 

ft 6,141,033; Michael , et al., issued October 31, 2000, entitled "Bandwidth Reduction of 

2| Multi-channel Images for Machine Vision". 

S7 Color or multi-spectral images are obtained using a color (or multi-spectral) 

m camera. Some color cameras use a prism with multiple dichroic coatings to split an 

19 image into multiple single-color images. Each single-color image is acquired 

20 simultaneously using a separate respective CCD or CMOS image sensor array. 

21 Although this method provides a high-resolution multi-color image at high speed, it is 

22 the most costly to implement. 

23 Another method for acquiring a color image is to sequentially filter the light from 

24 a scene using a rotating multi-color filter wheel and a single CCD or CMOS array to 

25 sequentially acquire each of the different single color images. Although this method 
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1 is less expensive, and provides a high-resolution color image, it is slow, and also 

2 suffers from the disadvantage that each single color (e.g., red green, or blue) image is 

3 acquired at a different time. 

4 A third method for acquiring a color image uses a single-chip CCD or CMOS 

5 sensor array. All three colors are acquired on the same sensor array at the same 

6 time. Although CMOS and CCD sensors detect the entire visible light spectrum (and 

7 sometimes the near-infrared as well), incident light must be separated into different 

8 spectral bands for color imaging. Separation is accomplished by depositing colored 
y filter materials (usually dyed or pigmented polymers) sequentially on the top surface 
jo of the sensor array, arranged in a mosaic pattern known as a color filter array (CFA). 
|l Photolithographic patterning of the different color layers produces the desired mosaic 
li array. 

J! Several CFA patterns can be used. A common arrangement is a four-pixel 

% repeat cell 100, called the "Bayer pattern". Each repeat cell 100 consists of two green 

|> (G), one red (R), and one blue (B) filter, as shown in Fig. 1A. This results in an 

16 alternating pattern of rows 110, wherein a first row consists of red and green 

17 alternating filters, and a second row consists of blue and green alternating filters, as 

18 shown in Fig. 1 B. Thus, in the Bayer pattern, the pixels are not evenly divided. In fact, 

19 there are as many green pixels as there are blue and red combined. 

20 Some single-sensor cameras can use alternatives to the Bayer filter pattern, 

21 such as repeating a repeating pattern of red, green, and blue stripes. Some single- 
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1 chip image sensors don't add up the different values of red, green and blue, but 

2 instead subtract values using the typesetting colors cyan, yellow, and magenta. 

3 One advantage of this method is that only one image sensor array is required 

4 to acquire a complete color image, and all the color information (e.g., red, green and 

5 blue) is recorded at the same moment. 

6 The raw output from a color image sensor with a Bayer filter pattern is a mosaic 

7 of red, green and blue (RGB) pixels of potentially different intensity. A single-chip color 

8 image sensor using an array with nxn pixel resolution is used to generate a three- 
(j? color RGB image with nxn pixels for each color plane (or color band). However, on 
ho the chip only 25% of the pixels are devoted to blue, and 25% of the pixels are devoted 
M to red, whereas 50% of the pixels are devoted to green. (The greater number of green 
Jfb, pixels is intended to simulate the relatively greater sensitivity of the human eye to 

m green light.) Therefore, the resolution of the raw data for each color plane is much 

if* less than nxn pixels. 

35 These lower resolution color plane images can be combined to create an 

16 apparently full-resolution color composite image through application of a Bayer 

17 pattern de-mosaic algorithm. This algorithm derives a red, green, and blue intensity 

18 value for each pixel location based on the values of the surrounding pixels. However, 

19 no new information is actually added by this step, since this is done by color 

20 "interpolation" from nearest-neighbor Bayer cells. For example, the red and blue 

21 signals at a green-sensitive pixel are estimated from the red or blue signal values on 
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1 neighboring red- or blue-sensitive pixels. This interpolation step is equivalent to up- 

2 sampling and low-pass filtering. 

3 Although less expensive than using a prism, and faster than sequentially 

4 acquiring multiple single-color images, this third approach to acquiring a color image 

5 yields an effectively low-resolution color image, since all colors are represented in a 

6 single color image. Consequently, using known methods, only low-resolution 

7 machine vision tasks can be performed using the color image obtained from a single- 

8 chip image sensor. 

5} High-resolution machine vision tasks require high-resolution image data. For 

|| example, to get highly accurate results when locating an image of an object using a 

Il search vision algorithm, high-resolution input image data is required. Due to the low 

fi resolution of each color plane of a single-chip color image sensor, a low-cost 

3E 

!fi standard color camera that incorporates such an image sensor is typically not used 

T": : 

SI for high-resolution machine vision applications. 
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2 Summary of the Invention 

3 In a general aspect of the invention, a method is provided for high-resolution 

4 (high-res) location of an object in an image using a single-chip color image sensor. 

5 In general, the sensor is sensitive to N bands of electromagnetic energy, and the 

6 sensor is characterized by a sensor geometry. The method includes: providing a 

7 high-res color model of an object, and acquiring a color image of an object with the 

8 color image sensor. An alignment location of the high-res color model is found in the 
S| color image so as to provide a single low-res location. Also, a true-phase high-res 
P color model having correct phase is obtained using the phase of the single low-res 
ll location, and using the sensor geometry. A high-res location of the true-phase high- 
# res color model is found in the acquired color image. 

jft In a preferred embodiment, finding the alignment location of the high-res color 

% model in the color image includes: providing N low-res single-band locations, one 

J$ low-res single-band location for each of the N bands; and combining the N low-res 

16 single-band locations into a single low-res location. 

n In a further preferred embodiment, combining the N low-res single-band 

18 locations includes: combining the N low-res single-band locations into a single low- 

19 res location using a weighted sum, each location being associated with a weighting 

20 factor. The weighting factor can incorporate a confidence factor. 

21 In a preferred embodiment, using the phase of the single low-res location, and 

22 using the sensor geometry, so as to provide a true-phase high-res color model 
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1 having correct phase includes: selecting one of N candidate high-res color models by 

2 using the phase of the single low-res location, or synthesizing a synthetic model. 

3 In a preferred embodiment, the sensor geometry is based on the Bayer pattern. 

4 In an alternate preferred embodiment, the sensor geometry is based on a three-color 

5 striped pattern. 

6 In yet another preferred embodiment, providing a high-res color model of an 

7 object includes: creating a high-res synthetic model of the object, or acquiring a 

8 high-res image of the object using a high-res camera, or acquiring a low-res image at 
tj each phase inherent in the sensor geometry. 

B In a further preferred embodiment, N is 3, and the bands represent red, green, 

jj and blue. The bands can also represent cyan, yellow, and magenta, or green, red, 

k and infra-red. N can also be 4, as when the bands represent blue, green, red, and 

|3 infra-red 

j| The method of the invention makes it possible to obtain high-resolution 

J5 performance from a standard image sensor, such as a single-chip image sensor. 

16 Since single-chip color image sensing is often the least expensive way to obtain color 

17 images, the invention helps to lower the cost of high-resolution machine vision 

18 performance. 

19 The method of the invention can improve the accuracy of machine vision 

20 analysis steps using a standard color camera by as much as a factor of two. 

21 The maximum difference between model and acquired image is half a pixel, 

22 assuming no scare difference and no rotation difference. 
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3 Brief Description of the Drawing 

4 The invention will be more fully understood by reference to the detailed 

5 description, in conjunction with the following figures, wherein: 

6 Fig. 1 A is a Bayer pattern cell; 

7 Fig. 1 B is a complete Bayer pattern of pixel filters including a plurality of the 

8 cells of Fig. 1A; 

H Fig. 2 is a flow chart illustrating a general method of the invention; 

|f Fig. 2A is a representation of the N separate low-resolution alignment steps for 

U each of the N bands, one each for R,G, and B; 

J§ Fig. 2B is a representation of a decomposition of a color image into a plurality 

§§ of single band images, each image being usable for object location; 
kk Fig. 3 is a representation of a high-resolution color image, wherein each pixel 

B includes RGB values; 

16 Fig. 3A is set of three pixel diagrams illustrating how a Bayer pixel array can be 

n treated as a low-res array of RGB pixels, or as a sparsely populated high-res color 

18 image, according to the invention; 

19 Fig. 4 is a representation of four sparsely populated high-res color images, one 

20 for each of four phases; 
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1 Fig. 5 is a representation of four possible desired low-resolution models, 

2 wherein each pixel is the average RGB value over a Bayer pattern cell of the high- 

3 resolution model of Fig. 4; 

4 Fig. 6 is a representation of a low-resolution color image constructed from a 

5 high-resolution color image; and 

6 Fig. 7 is a representation of four sub-pixel phases, and the location of the sub- 

7 pixel phases in relationship to a pixel grid. 

8 

H Detailed Description 

M Referring to Fig. 2, the method of the invention includes first providing a high 

M resolution color model of an object 200. This step 200 involves obtaining R, G, B (red, 

W3 green, blue) or other color band values for each pixel in the image, as shown in Fig. 3, 

|4 that will be part of the alignment model at high resolution, where high resolution 
means the resolution of a multi-chip color image sensor wherein each chip has the 

16 same dimensions, and has the same number of pixels, as the single-chip sensor; 

17 and constructing the alignment model from that high resolution image. The specifics 

18 of the alignment model are known in the art, and depend on the technique for 

19 alignment For example, for a correlation-based alignment technique, the model may 

20 consist of pixel values. For a geometric-based alignment technique, the model may 

21 consist of geometric descriptions. 
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1 Obtaining the values in the color bands can be done in several ways. One way 

2 is by acquiring an image of the object using a multi-sensor imager. The resulting 

3 acquired image is illustrated by Fig. 3. Note that each pixel 300 has full color band 

4 information. The high-resolution color alignment model is constructed from that 

5 acquired high-resolution image, as explained above. 

6 A second way is to synthetically generate the color bands and consequently the 

7 high-resolution color alignment model from CAD data, or from a synthetically 

8 rendered high resolution image. 

2> A third way is to use a single chip sensor and acquire multiple images of the 

jffl object at all possible phases, such as illustrated by Fig 4. For the bayer pattern, this 

|i is accomplished by translating either the camera or the object by one pixel in the x- 

4S direction, next by one pixel in the y-direction, and finally by one pixel in the x-direction 

Irs and one pixel in the y-direction compared with the initial alignment of camera and 

14 object. Note that the position of the object will have moved by one pixel and the 

If location will need to be adjusted. 

16 The multiple images of the object at all possible phases can be combined into 

17 a single high-resolution color image by copying the missing values from phases that 

18 are not missing those values. Doing this is a two step process: step one is to detect 

19 the phase from multiple image acquisitions. This can be done by assigning "phase 1" 

20 to the first acquired image, and by aligning the subsequent images to that first image, 

21 and then using Fig. 7 to assign the subsequent images to phases 1, 2, 3 and 4 (702). 

22 ff the camera or object motion can be very accurately controlled so that single pixel 
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1 motions can be easily obtained, alignment will not be necessary. Step two is to 

2 combine the images corresponding to the phases into a single high resolution 

3 image, as shown in Fig. 3. 

4 Referring to Figs. 3 and 4, to illustrate this process, it is helpful to consider an 

5 example. An example is the upper left pixel 300 which consists of values Ra, Ga, Ba. 

6 It is the combination of the upper left pixel Ra 403 from the image 402 associated with 

7 phase 1 of Fig. 4, and the upper left pixel Ga 405 from the image 404 associated with 

8 phase 2, and Ba 409 from the image 408 associated with phase 4. Note that Ga 
h$ could also have been obtained from Ga 407 from the image 406 associated with 

15 phase 3 of Fig. 4, or alternatively, Ga could have been obtained as the average of Ga 
J 405 and Ga 407. The high-resolution color alignment model can be constructed from 
t£ this single high-resolution color image. 

pj Referring again to Fig. 2, in step 202, a color image of an object is acquired 

M with a color image sensor, such as a single chip color image sensor. The color 

S3 image can be treated in two different ways, as illustrated by Fig. 3A. 

16 A first way is to treat the color image acquired with a single-chip color image 

17 sensor as a low-resolution color image 330 as shown in Fig. 3A. The first way is 

18 known in the art, and is how single-chip color image sensors are conventionally used. 

19 Color pixel values at locations in between spatial sampling locations (the R, G, B 

20 values of the CFA) can be interpolated, and the image spatial frequencies are 

21 assumed to follow the requirements for interpolation. (Note that interpolation is 

22 equivalent to upsampfing and fow pass filtering.) 

Cognex Confidential 1 0 1 2/29/0 1 



C01-016 

1 As recognized by the invention, a second way to treat the color image acquired 

2 with a single-chip color image sensor is to handle the color image as N sparsely 

3 sampled single-band images, as illustrated by array 320 in Fig. 3A„ where in the case 

4 of an RGB color image, N=3. Each single band image is at high resolution, but 

5 sparsely sampled. In other words, most (2/3) of the pixel components R, G, B (red, 

6 green, blue) are missing at each spatial location in the image. 

7 Referring again to Fig. 2, in the step of finding the alignment of the high-res 

8 color model 204, the color image is treated as a low resolution color image wherein 
H the pixel values between sampled colors have been interpolated or equivalently, that 
W each band of the color image is low resolution. We find the location of the object in 
|i this low-resolution color image by finding the location of the object in each band 

J independently, as illustrated by Fig. 2A, and combining the resulting locations to 

m provide a single low-resolution location. 

ru 

m Specifically, a red image 210 is aligned 222 using a red model 216 so as to 

% provide a location to a combiner 228. Likewise, a green image 212 is aligned 224 

16 using a green model 218 so as to provide a location to the combiner 228, and a blue 

17 image 214 is aligned 226 using a blue model 220 so as to provide a location to the 

18 combiner 228. The combiner 228 then provides a single low-resolution location 230. 

19 Fig. 2B illustrates how a color image 240 can be decomposed into N single- 

20 band images 250, 260, 270 that can each be separately used for determining object 

21 location. Using the high-resolution model to find an object in a low-resolution single- 

22 band image is done by reducing the resolution of the single-band version of that high- 
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1 resolution model. One way of doing this is to reduce the resolution of the image used 

2 to train the high resolution model, as shown for example in Fig. 6 where the low- 

3 resolution image is one quarter the resolution of the high-resolution model, where 

4 each low-resolution pixel 600 includes an average R,G, and B value as shown. 

5 Next, the low-resolution model is trained from the low-resolution image, 

6 resulting in a desired low-resolution model 502,504,50,508 or Fig. 5, depending on 

7 the phase, as yet unknown. Another way of reducing the resolution of the model is to 

8 directly reduce the resolution of the model itself, as is known in the art. 

H The found locations of the object in each band can be combined in a plurality of 

Si ways, as represented by the combiner 228 in Fig. 2A. One way is to use a weighting 

U factor for each found location. The weighting factor can represent relative confidence 

h in the associated found location. The found locations are each multiplied by an 

la associated weighting factor, and these terms are added together to provide the single 

y. low-res location. The weighting factors can be normalized, such that the sum of all of 

g the weighting factors can be 1 , for example. 

16 Another way to combine the found locations is to use a geometric mean of the 

17 locations. Alternatively, an arithmetic mean can be used. Also, the location 

18 associated with a maximum confidence can be selected as the single low-res 

19 location. 

20 Referring again to Fig. 2, the step of providing a true-phase high-res color 

21 model 206 involves comparing the single low-res location with four possible sub-pixel 

22 phases 702, fabefed p1 , p2, p3, and p4 in the context of a pixel grid 700 of Fig. 7. The 
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1 closest of the four phases is associated with the most representative high-resolution 

2 model of the four high-res models 402,404, 406, 408 shown in Fig. 4. In an 

3 embodiment where the sensor geometry involves only three phases, such as the 

4 RBG stripe geometry, there would be only three high-res models. In general, there 

5 will be as many high-res models as there are phases inherent in the sensor 

6 geometry. 

7 Referring again to Fig. 2, in the step of finding the high-res location of the true- 

8 phase high-res color model in the acquired color image 208, it is now possible to 

H perform high-resolution alignment. Here, the color image provided by the single-chip 

3 color image sensor is treated as a high-resolution color image, albeit a sparsely 

lf( sampled color image. Thus, using the true-phase high-res color model to search the 

jj sparsely sampled high-resolution color image that was originally acquired by the 

u single-chip color image sensor, a high-resolution location of the true-phase high-res 

W color model can be found. 

ji Note that this step is probably only correcting the results from the low- 

16 resolution alignment by less than 1 pixel. Consequently, it may be possible to 

17 optimize the alignment algorithm by reducing the area to be searched. 

18 Also, note that nothing in this approach limits alignment to translation only. For 

19 example, it can be used for translation, rotation, scale, general affine transformation 

20 or perspective alignment as well. 

21 Other variants and embodiments will occur to those skilled in the art, without 

22 departing from the spirit and scope of the invention. Accordingly, the invention is not 
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1 intended to be limited by the detailed description, except as set forth in the following 

2 claims. 
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